Builds on Count Vectorisation by normalising the document frequency measure by the overall corpus frequency. Common words receive a large penalty:
\[
W(t,d) = TF(t,d) / log(N/DF_{t})
\]
For example:
If the term ‘cat’ appears 3 times in a document of 100 words then Term Frequency given by: \(TF(t,d)=3/100\), and
If there are 10,000 documents and cat appears in 1,000 documents then Normalised Document Frequency given by: \(N/DF_{t}=10000/1000\) so the Inverse Document Frequency is \(log(10)=1\),
Three input texts with a distance weighting (\(d/2\), where \(d<3\)):
the cat sat on the mat
the cat sat on the fluffy mat
the fluffy ginger cat sat on the mat
fluffy
mat
ginger
sat
on
cat
the
fluffy
1
1
0.5
0.5
2.0
mat
0.5
1.5
ginger
0.5
0.5
1.0
1.5
sat
3.0
3.0
2.5
on
1.5
3.0
cat
2.0
the
How Big is a TCM?
The problem:
A corpus with 10,000 words has a TCM of size \(10,000^{2}\) (100,000,000)
A corpus with 50,000 words has a TCM of size \(50,000^{2}\) (2,500,000,000)
Cleaning is necessary, but it’s not sufficient to create a tractable TCM on a large corpus.
Enter Document Embeddings
Typically, some kind of 2 or 3-layer neural network that ‘learns’ how to embed the TCM into a lower-dimension representation: from \(m \times m\) to \(m \times n, n << m\).
Similar to PCA in terms of what we’re trying to achieve, but the process is utterly different.
Sentiment Analysis
Requires us to deal in great detail with bi- and tri-grams because negation and sarcasm are hard. Also tends to require training/labelled data.
